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ABSTRACT 


This essay discusses the emergence of the program evaluation dissertation in our doctoral program as the 
preferred dissertation in practice (DiP) option. We also outline important considerations that must be reviewed 
with students when considering this approach. Our students are professional educational leaders in the settings 
where they conduct their dissertation research, emphasizing the importance of our doctoral faculty in 
addressing the potential implications of blending dissertation research with professional practice. Using the 
utilization-focused evaluation approach as a framework, we address professional and practical considerations 
to ensure effective evaluation designs to examine a specific problem of practice. These considerations include 
the student's relationship to the program being evaluated, the impact on their professional position, support for 


the evaluation, access to data collection, potential consent concerns, and the utilization of findings. We 
conclude with additional ethical considerations to be considered when supporting program evaluation 


dissertation work. 
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For the past 20 years, our school improvement doctoral 
program at a state-funded comprehensive university has trained 
many educators to initiate and sustain significant change in their 
educational settings. Since educational leaders are tasked with 
generating and using data to inform their practice in various ways 
(Bauer & Brazer, 2012), we have experimented with various 
dissertation formats. The traditional five-chapter dissertation has 
been the most prevalent format that students use, primarily due to 
faculty preference. This approach is utilized in both Ph.D. and Ed.D. 
programs and mirrors the scientific method process, but is also used 
for qualitative research. This format includes chapters pertaining to 
introduction/rationale, review of literature, methods, results, and 
interpretations/recommendations (Calabrese, 2006). Despite the 
constant demand for educational data, it is argued that educational 
environments rarely use research-based information to inform their 
practice (Dagenais et al., 2012). 


In response to doctoral student feedback, our program piloted a 
program evaluation dissertation format. This format is becoming 
increasingly popular within educational research since program 
evaluation is crucial to educational initiatives and even 
recommended by the United States Department of Education (U.S. 
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Department of Education, 2014). As Stewart (2016) noted, this trend 
will continue as schools become increasingly reliant on federal 
funding that requires evaluative components. The program 
evaluation dissertation format is also of interest to the doctoral 
students in our school improvement Ed.D. program involved in 
designing, implementing, or evaluating various problems of practice 
in their real-world capacities. 


Utilization-Focused Evaluation Process 


We strive to have students use a process similar to the 
utilization-focused evaluation (U-FE) process (Patton, 2012) when 
initially outlining their program evaluation dissertation proposals. 
UFE is based on the principle that evaluations should be useful both 
during the evaluation process and after findings are generated. U-FE 
fits well within the Ed.D. philosophy and the dissertation in practice 
(DiP) approach because it is personal and situational. The evaluation 
context is significant, and findings are expected to be used in 
practical ways that make sense for the program. There are 17 steps 
to the U-FE process that begins with assessing and building capacity 
for the evaluation. The U-FE process ends with a focus on the use of 


This journal is supported by the Carnegie Project on 
the Education Doctorate: A Knowledge Forum on the 
EdD (CPED) cpedinitiative.org 


impactinged.pitt.edu 
Vol. 7 No. 1 (2022) 


ISSN 2472-5889 (online) 
DOI 10.5195/ie.2022.226 


16 


Professional and Practical Considerations 


findings (Patton, 2012). As we explain the professional and practical 
considerations we address with students, we also outline their 
connection to specific steps in the U-FE process and the CPED DiP 
framework. 


Connection to the Dissertation in Practice (DiP) 


Since program evaluation is inherently collaborative, this DiP 
approach fits well with the Carnegie Project on the Education 
Doctorate (CPED) scholarly practitioner focus in two ways. First, 
students are more engaged with reflective problem-solving by their 
interactions with various program stakeholders. Secondly, this 
process guides students through various systematic complexities 
specific to the program context. The DiP approach allows students to 
more deeply and holistically examine their problems of practice in a 
way that may not be available in traditional research methods. 


While the program evaluation approach works well with the DiP 
philosophy, there are professional and practical considerations that 
faculty must address with their doctoral students before 
implementation. This manuscript outlines these considerations from 
both the faculty and student perspective and discusses how these 
conversations facilitate student development into their role as 
scholarly practitioners in an educational evaluator capacity. 


PROFESSIONAL CONSIDERATIONS 


The professional considerations we address with students 
speak to their dissertation work in an educational leadership 
capacity. As with most Ed.D. programs, our students are full-time 
practitioners, often in school, district, or state-level leadership 
positions. We specifically discuss their role with the programs they 
seek to examine. As a highly regarded best practice, our program 
encourages students to focus their dissertation research on a topic in 
which they are passionate (Butin, 2010). For our program, the topic 
should center on identifying a problem of practice related to school 
improvement. Students who choose the program evaluation 
dissertation often seek to evaluate programs housed in their school 
environments. Their evaluations are not limited to DiPs and can 
involve a range of programs they would like to investigate with the 
intention of school improvement. However, professional and ethical 
concerns arise when our students are in positions of authority or 
obligation directly related to the program or its stakeholders. 
Therefore, before the student begins their proposal, we ask them the 
following questions: 


1. What is your relationship with this program? 


2. Will the findings of your evaluation impact your professional 
position? 
3. Is your school/district/state supportive of evaluating this 
program? 
These questions align with the first three U-FE tasks centered on the 
program, organizational, and evaluator readiness for a utilization- 
focused evaluation and engaging the appropriate individuals in the 
process (Patton, 2012). Answers to these questions can immediately 
alter the program evaluation dissertation's trajectory in the design 
and implementation stages, as described below. 


Student Relationship to the Program 


In our initial dissertation conversations, the first consideration 
we outline is the student's relationship to the program they seek to 
evaluate. For example, STUDENT AUTHOR initially wanted to 
collect data on the Autism Support Group he developed and led ten 
years ago. While the support group is now sustained by volunteers, 
STUDENT AUTHOR still attends the meetings periodically and has a 
presence within the support group (Author and Author). During 
conversations with his committee (AUTHOR and AUTHOR), it was 
agreed that his direct relationship with the program could cause a 
conflict of interest when interviewing participants and conducting 
focus groups. As aresult, STUDENT AUTHOR enlisted a research 
assistant to collect data for this program evaluation dissertation. 
STUDENT AUTHOR also performed member checks on his data 
analysis (Merriam & Tisdell, 2015) to increase the trustworthiness 
and dependability of his analysis. This additional layer of analysis 
allowed participants to ensure STUDENT AUTHOR accurately 
interpreted their experiences in the program. 


Impact on Professional Positions 


In addition to their relationship with the program, we also 
discuss the potential impact the evaluation could have on the 
students’ professional position. If our student's professional 
performance is tied to a program's outcomes, we consider the 
implications and various ways to approach these concerns. For 
example, instead of examining an entire program that a student 
oversees in a professional capacity, we may recommend that the 
student evaluate one portion of the program that another individual 
administers. This approach does not always work well, depending on 
the evaluation needs, but it can be an option for students who need a 
safeguard. While our goal is to help our students develop and grow 
into strong scholarly practitioners who initiate and sustain school 
improvement, we also strive to respect and protect them 
professionally. 


Program Evaluation Support 


Professionalism is also tied to the support needed for the 
program evaluation dissertation. Early in dissertation discussions, we 
ask students about the anticipated support they will receive from 
their school, district, state, etc. We also ask about the potential 
findings. 


PRACTICAL CONSIDERATIONS 


Once the professional considerations for the program 
evaluation dissertation are discussed, we outline potential practical 
considerations. Since our doctoral students are practitioners, these 
considerations are important to discuss early in the dissertation 
process. As experienced educators and evaluators, faculty must 
guide students through thoughtful concerns for their dissertation 
research requirements, as well as their school/educational 
environments. Given the political nature of some program 
evaluations, the questions need to be considered from a variety of 
angles. The student is encouraged to investigate background 
information on the program, such as when the program began, who 
started the program, who or which departments support and monitor 
the program, and if there are any potential issues with evaluating the 
program in advance, as student success should be at the forefront of 
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the dissertation. Students may need guidance and support in order 
to navigate challenging situations. 


Example questions we ask students include: 


1. Will you be able to collect the data you need to effectively 
evaluate the program? 


2. Are there any anticipated issues with gaining consent, 
assent, parental permission, etc.? 


3. How will findings from this evaluation be used? 


These questions align with U-FE tasks 5-17 that begin with outlining 
the evaluation's intended uses and priorities and end with the use of 
findings for improvement purposes (Patton, 2012). Answers to these 
questions may significantly affect the data collection process and the 
use of findings. 


Effective Program Evaluation Data Collection 


Research in educational settings, particularly public schools, 
can be cumbersome and difficult to navigate. Schools, districts, and 
states have varying policies and procedures for conducting research. 
The recent emergence of COVID-19 has further complicated these 
processes. Due to these barriers, we ask students to outline multiple 
data collection plans. For example, Dr. STUDENT AUTHOR initially 
planned on face-to-face focus groups to examine his Autism Support 
Group. He planned a backup option to utilize an online platform for 
the focus groups, as well as to evaluate another similar support 
group. Due to COVID-19, he was forced to use the virtual option of 
data collection. Regardless of the dissertation approach, we 
encourage all doctoral students to consider creating contingency 
options and plans and incorporate them into their initial IRB 
application materials to ensure continuity in data collection. 


Although Dr. STUDENT AUTHOR was fortunate to have 
additional data collection options available during his dissertation 
process, additional limitations may arise specific to program 
evaluation dissertations. For example, programs can be limited in the 
number of participants available for data collection. Therefore, Dr. 
STUDENT AUTHOR had another data collection contingency plan 
specific to participants. Since the emergence of COVID-19 left many 
things unpredictable, his second option was to conduct focus groups 
with a similar program in a neighboring area. While this option would 
still provide beneficial information for his evaluation, it would not 
have been specific to the support group he developed and led. 


Permissions and Consents 


Since an educational setting's culture can significantly impact 
the evaluation and research efforts, considerations about consent 
must be discussed. This is an important practice where students 
combine their schools' practical knowledge into their emerging 
knowledge as scholars. For example, students who know their target 
population is largely non-English speaking will outline thoughtful 
approaches to articulating evaluation efforts, translating consents, 
and modifying data collection tools as needed. 


Utilization of Evaluation Findings 


The final practical consideration we discuss with students is the 
utilization of the evaluation. As we strive to have students use the 
utilization-focused evaluation approach (Patton, 2012), this requires 
that stakeholders commit to using the evaluation findings to improve 


the practice in some capacity. The utilization of findings also 
connects to the appeal of the program evaluation approach to our 
doctoral students. The practical nature of this work is meaningful for 
our students who are passionate about improving schools and 
appears to strengthen their identities as scholarly practitioners. 


Since our dissertations are publicly accessed, conversations 
about confidentiality and permissions are critical (Patton, 2012). 
Masking educational settings or participants’ identities is always 
practiced in our program and often sufficient; however, we require 
students to have conversations with stakeholders about potential 
negative, unwelcome, or unanticipated evaluation findings. As 
STUDENT AUTHOR shared in his post-dissertation experience, he 
highlights that "students should remember that program evaluations 
are effective at answering questions within organizations and that the 
answers to their questions may reveal problems or truths that are not 
easily accepted." We have had instances of students not being 
granted access to externally evaluate programs in schools due to 
stakeholder's concerns with data and the results of the program 
evaluation. 


As in traditional program evaluations, stakeholders connected 
to the program often request updates or debriefs as programs are 
being examined. We discuss the context of their situation and the 
optimal feedback process. For example, sharing preliminary findings 
with important stakeholders can prepare them for the results in final 
reports (Patton, 2012). If the program a student is evaluating is 
highly sensitive, visible, political, or controversial, we encourage 
them to have regular conversations with the appropriate individuals. 


CONTINUED CONSIDERATIONS 


As we continue to implement the program evaluation 
dissertation in practical settings using the U-FE approach (Patton, 
2012), we have outlined six specific professional and practical 
considerations to address with our students; however, additional 
questions and considerations remain. As our students work as full- 
time practitioners and emerging scholarly practitioners examining 
important problems of practice, continued discussions around ethics 
in program evaluation are necessary. Faculty in our program have 
posed the following questions: 


Is it ethical to conduct a program evaluation in one’s own 
educational environment as an internal evaluator? 


How can we ensure "no harm" and avoid coercion when 
EdD students are evaluating students, teachers, and staff 
they oversee? 


Should EdD students only be allowed to conduct external 
program evaluations, as opposed to internal program 
evaluations? 


These questions will continue to be discussed as we work with 
students through their program evaluation dissertations while helping 
them balance the reality of their practitioner work. Similar to the ever- 
changing landscape of education, our work with doctoral students 
must adapt as well. We may never have a perfect program 
evaluation dissertation model that fits with all students and their 
circumstances. Challenges and barriers will continue to present 
themselves in various ways. Having individual conversations with 
students to guide them through professional and practical 
considerations in their dissertation will strengthen their work and their 
scholarly practitioner identities. 
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